1. Field of the Invention
This invention relates generally to image sensor array processing. More particularly, this invention relates to circuits and methods for adjusting resolution of image sensors. Even more particularly, this invention relates to circuits and methods for adjusting resolution of image sensors by decimating the addressing of the image sensors into sub-groups of the array of the image sensor, averaging the columns of each of the sub-groups of the image sensor, and selectively averaging in a high intensity light environment or binning in a low intensity light environment of multiple rows of the average of the columns of the sub-group of the array of the image sensors.
2. Description of Related Art
Digital Cameras employing CMOS image sensor technology include image processing and JPEG (Joint Photographic Experts Group) compression for adjusting the resolution of the camera. In general, the image sensor operates in several modes. It takes full resolution image in a relative lower speed (1 to 15 frames per second depending on the image format) which is stored in a memory. The image sensor must also acquire low resolution images at high speed (about 30 frames per second) for viewfinder or short video. In most of the CMOS image sensor designs, low resolution high speed images are acquired by decimation or partitioning the image array in to groups of pixels and choosing a sub-set of the group of pixels to sub-sample a sub-set of pixels within the group of pixels that has been selected to represent the whole image.
FIGS. 1a and 1b illustrate the sub-sampling of an array of Bayer pattern configured Complementary Metal Oxide Semiconductor (CMOS) Active Pixel Sensors (APS). The Bayer pattern, as shown in U.S. Pat. No. 3,971,065 (Bayer), describes a format for a color filter array. In the array as shown, the Bayer pattern has four sensors arranged in a two by two matrix of CMOS APS's. The CMOS APS's receive the Red, Green and Blue of the standard color video construction. One Pixel receives the Red, one the Blue, and the remaining two pixels receive the Green and are designated red (R), green-1 (G1), green-2 (G2), and blue (B).
In FIG. 1 the array is structured to illustrate a 3:1 ratio sub-sampling on the Bayer pattern. The array 5 of CMOS APS's shows a 6×6 array of the Bayer pattern sensors. The array 5 is physically an array having 12 pixels in the horizontal dimension and 12 pixels in the vertical dimension. The Bayer pattern groups these pixels into the 2×2 groups (Red, Green-1, Green-2, and Blue) of pixels. The sub-sampling then further groups the pixels according to the ratio of the sub-sampling. Thus, each sub-group 7a, 7b, 7c, and 7d has a 6×6 array of CMOS APS's that is further divided into a 3×3 Bayer pattern. In a sub-sampling, a central Bayer grouping of each sub-group 7a, 7b, 7c, and 7d is chosen as the output pixels RO, G1O, G2O, and BO of the array.
In general, the output pixels as a function of original image pixel (not considering the fixed spatial offset) information are given by:RO(k,l)=[R(2×n×k, 2×n×l)]G1O(k,l)=[G1(2×n×k+1, 2×n×l)]G2O(k,l)=[G2(2×n×k, 2×n×l+1)]BO(k,l)=[B(2×n×k+1, 2×n×l+1)]  (1)                where:                    n is the decimation ratio of the sub-sampling of the array.            k is the counting variable for a row dimension of the sub sampled array 15.            l is the counting variable for the column dimension of the sub sampled array 15.            RO is the red pixel of the sub sampled array 15.            G1O is the first green pixel of the sub sampled array 15.            G2O is the second green pixel of the sub sampled array 15.            BO is the blue pixel of the sub sampled array 15.                        
Pixel sub-sampling reduces the output bandwidth that the frame rate can be increased with same pixel readout speed. However, the drawback of pixel sub-sampling is the lost of spatial resolution that will introduce aliasing to the image. In additional, the image obtained from pixel sub-sampling has a very poor quality at low light level because of the effective small sensing area.
The images sensors are increasing in size to accommodate the image formats such as the Super Extended Graphics Array (SXGA) display specification that is capable of displaying 1280×1024 resolution, or approximately 1.3 million pixels or the Quantum Extended Graphics Array (QXGA) display specification that is capable of supporting 2048×1536 resolution, or approximately 3.2 million pixels. As the image sensors become larger, and decimation ratio becomes higher, more and more image information will be lost due to pixel sub-sampling.
To enhance the spatial resolution of decimated image, pixel binning and/or averaging is desired. Thus, the output pixels RO, G1O, G2O, and BO of the array 15 will represent all the information of its neighboring pixels of the sub-group 7a, 7b, 7c, and 7d of the original array 5 of CMOS APS's. In general, for the n×n pixel binning, the value of output pixels RO, G1O, G2O, and BO are:
                                                        R              O                        ⁡                          (                              k                ,                l                            )                                =                                    ∑                              i                =                0                                            n                -                1                                      ⁢                                          ∑                                  j                  =                  0                                                  n                  -                  1                                            ⁢                              [                                  R                  ⁡                                      (                                                                                            2                          ×                          n                          ×                          k                                                +                                                  2                          ×                          i                                                                    ,                                                                        2                          ×                          n                          ×                          l                                                +                                                  2                          ×                          j                                                                                      )                                                  ]                                                    ⁢                                  ⁢                              G            ⁢                                                  ⁢                          1              O                        ⁢                          (                              k                ,                l                            )                                =                                    ∑                              i                =                0                                            n                -                1                                      ⁢                                          ∑                                  j                  -                  0                                                  n                  -                  1                                            ⁢                              [                                  G                  ⁢                                                                          ⁢                  1                  ⁢                                      (                                                                                            2                          ×                          n                          ×                          k                                                +                                                  2                          ×                          i                                                +                        1                                            ,                                                                        2                          ×                          n                          ×                          l                                                +                                                  2                          ×                          j                                                                                      )                                                  ]                                                    ⁢                                  ⁢                              G            ⁢                                                  ⁢                          2              O                        ⁢                          (                              k                ,                l                            )                                =                                    ∑                              i                =                0                                            n                -                1                                      ⁢                                          ∑                                  j                  =                  0                                                  n                  -                  1                                            ⁢                              [                                  G                  ⁢                                                                          ⁢                  2                  ⁢                                      (                                                                                            2                          ×                          n                          ×                          k                                                +                                                  2                          ×                          i                                                                    ,                                                                        2                          ×                          n                          ×                          l                                                +                                                  2                          ×                          j                                                +                        1                                                              )                                                  ]                                                    ⁢                                  ⁢                                            B              O                        ⁡                          (                              k                ,                l                            )                                =                                    ∑                              i                =                0                                            n                -                1                                      ⁢                                          ∑                                  j                  =                  0                                                  n                  -                  1                                            ⁢                              [                                  B                  ⁡                                      (                                                                                            2                          ×                          n                          ×                          k                                                +                                                  2                          ×                          i                                                +                        1                                            ,                                                                        2                          ×                          n                          ×                          l                                                +                                                  2                          ×                          j                                                +                        1                                                              )                                                  ]                                                                        (        2        )                            where:                    n is the decimation ratio of the sub-sampling of the array.            i is the counting variable for the neighboring pixels in a row dimension of the sub sampled array 15.            j is the counting variable for the neighboring pixels for a column dimension of the sub sampled array 15.            k is the counting variable for a row dimension of the sub sampled array 15.            l is the counting variable for the column dimension of the sub sampled array 15.            RO is the red pixel of the sub sampled array 15.            G1O is the first green pixel of the sub sampled array 15.            G2O is the second green pixel of the sub sampled array 15.            BO is the blue pixel of the sub sampled array 15.                        
Similarly, for n×n pixel averaging, the value of output pixels RO, G1O, G2O, and BO are:
                                                                                          R                  O                                ⁡                                  (                                      k                    ,                    l                                    )                                            =                                                1                                      n                    ×                    n                                                  ⁢                                                      ∑                                          i                      =                      0                                                              n                      -                      1                                                        ⁢                                                            ∑                                              j                        =                        0                                                                    n                        -                        1                                                              ⁢                                          [                                              R                        ⁡                                                  (                                                                                                                    2                                ×                                n                                ×                                k                                                            +                                                              2                                ×                                i                                                                                      ,                                                                                          2                                ×                                n                                ×                                l                                                            +                                                              2                                ×                                j                                                                                                              )                                                                    ]                                                                                                                                                                                                                          G                      ⁢                                                                                          ⁢                                              1                        O                                            ⁢                                              (                                                  k                          ,                          l                                                )                                                              =                                        ⁢                                                                  1                                                  n                          ×                          n                                                                    ⁢                                                                        ∑                                                      i                            =                            0                                                                                n                            -                            1                                                                          ⁢                                                  ∑                                                      j                            -                            0                                                                                n                            -                            1                                                                                                                                                                                                                                    ⁢                                          [                                              G                        ⁢                                                                                                  ⁢                        1                        ⁢                                                  (                                                                                                                    2                                ×                                n                                ×                                k                                                            +                                                              2                                ×                                i                                                            +                              1                                                        ,                                                                                          2                                ×                                n                                ×                                l                                                            +                                                              2                                ×                                j                                                                                                              )                                                                    ]                                                                                                                                                                                                                                                  G                        ⁢                                                                                                  ⁢                                                  2                          o                                                ⁢                                                  (                                                      k                            ,                            l                                                    )                                                                    =                                            ⁢                                                                        1                                                      n                            ×                            n                                                                          ⁢                                                                              ∑                                                          i                              =                              0                                                                                      n                              -                              1                                                                                ⁢                                                      ∑                                                          j                              =                              0                                                                                      n                              -                              1                                                                                                                                            ⁢                                                                                                                                                                                                      ⁢                                          [                                              G                        ⁢                                                                                                  ⁢                        2                        ⁢                                                  (                                                                                                                    2                                ×                                n                                ×                                k                                                            +                                                              2                                ×                                i                                                                                      ,                                                                                          2                                ×                                n                                ×                                l                                                            +                                                              2                                ×                                j                                                            +                              1                                                                                )                                                                    ]                                                                                                                                                                                                                                                                            B                          O                                                ⁡                                                  (                                                      k                            ,                            l                                                    )                                                                    =                                            ⁢                                                                        1                                                      n                            ×                            n                                                                          ⁢                                                                              ∑                                                          i                              =                              0                                                                                      n                              -                              1                                                                                ⁢                                                      ∑                                                          j                              =                              0                                                                                      n                              -                              1                                                                                                                                            ⁢                                                                                                                                                                                                      ⁢                                          [                                              B                        ⁡                                                  (                                                                                                                    2                                ×                                n                                ×                                k                                                            +                                                              2                                ×                                i                                                            +                              1                                                        ,                                                                                          2                                ×                                n                                ×                                l                                                            +                                                              2                                ×                                j                                                            +                              1                                                                                )                                                                    ]                                                                                                                              (        3        )                            where:                    n is the decimation ratio of the sub-sampling of the array.            i is the counting variable for the neighboring pixels in a row dimension of the sub sampled array 15.            j is the counting variable for the neighboring pixels for a column dimension of the sub sampled array 15.            k is the counting variable for a row dimension of the sub sampled array 15.            l is the counting variable for the column dimension of the sub sampled array 15.            RO is the red pixel of the sub sampled array 15.            G1O is the first green pixel of the sub sampled array 15.            G2O is the second green pixel of the sub sampled array 15.            BO is the blue pixel of the sub sampled array 15.                        
Equations (1), (2), and (3) indicate that the pixel sub-sampling has the lowest spatial resolution and no signal level enhancement. Pixel binning has the high spatial resolution with highest signal level enhancement (factor of n2). Pixel averaging has the high spatial resolution, but without the signal level enhancement.
Each of the different image decimation techniques of CMOS APS's (image sub-sampling, image binning, and image averaging) have their own set of advantages and disadvantages.
In image sub-sampling no analog circuit modification is required within the CMOS image sensor. A digital control circuit manipulates the sub-sampling addresses during the readout. For an n:1 image reduction ratio, the output rate at which the imaged is transferred from the array 15 is reduced to 1/n2.
In image binning, binning processing is either the digital domain or analog domain. For image binning in digital domain, an on-chip analog-to-digital converter converts all the pixel signals to digital values and store the values in a static random access memory (SRAM). Then, the stored pixel values are added digitally based on the color and number of pixels in the reduction window. This approach requires that the transfer rate of the pixel values from the SRAM to be at a higher speed (full resolution at 30 frames per second). This further requires that the SRAM to be relatively very large. If the CMOS APS's array, the analog-to-digital-converter, and the SRAM are integrated on the same substrate, the substrate dissipates very high power and is very large. Image binning in the analog domain, increases the complexity of analog circuit design significantly to accomplish the real time pixel binning.
A simple image averaging can be done by changing the column sample/hold circuit design. However, although pixel averaging gives the good spatial resolution, signal level at low light illumination condition still results the poor image performance.
FIG. 2 shows a typical CMOS Active Pixel Sensor (APS) of the prior art, using a photo-diode as a photo-conversion device for example. The drain terminals of the transistors M1 and M2 are connected to the power supply voltage distribution line, VDD The source of the transistor M2 is connected to the anode of the photo-diode DF. The cathode of the photo-diode is connected to the ground reference point. The capacitance CFD is the inherent capacitance of the photo-diode DF.
The gate of the transistor M2 is connected to a reset terminal to receive the reset signal Vrst. The sensor readout node FD, that is the anode of the photo-diode DF, is first reset to a high voltage level (VDD) by changing the reset signal Vrst from a low voltage level (0) to a high voltage level (VDD) to charge the capacitance CFD. At the completion of charging the capacitance CFD, the reset signal Vrst is changed from the high voltage level (VDD) to the low voltage level (0). Since light is shining on the photo-diode DF, photo-generated electrons are collected at node FD and the voltage at the node FD decreases in the process. At the end of the exposure duration the voltage at node FD is measured, thus completing one photo-sensing cycle. The photo-sensing cycle is completed by activating the transistor M3 by changing the row select signal from the low voltage level (0) to the high voltage level (VDD) that reads the differential voltage of signal and reset level to column sample/hold circuit (S/H CKT).
The gate of the transistor M1 is connected to the node FD and the source of the transistor M1 is connected to the drain of the transistor M3. The transistor M1 acts as a source follower such that the voltage present at the source of the transistor M1 “follows” directly the voltage present at the gate of the transistor M1 and is one transistor threshold voltage VT below the voltage present at the gate of the transistor M1.
The gate of the transistor M3 is connected to the row select line to receive the row select signal Vrow. The source of the transistor M3 is connected to the sample and hold circuit. The sample and hold circuit provides the pixel output voltage VOUT to the column bus ColBus. The column bus ColBus interconnects all the APS's present on a column of an array of APS's. When the row select signal changes from a low voltage level (0V) to a high level (VDD), the transistor M3 turns-on and the voltage present at the source of the transistor M1 is transferred to the output of the APS to couple the voltage that is proportional to the intensity of the light L. The output signal Vout—pixel of the APS is coupled to sample and hold circuit for conditioning and control for transfer to the column bus ColBus and to the video amplifier for further conditioning and readout.
The column sample and hold circuit, as shown in FIG. 2 is shown in more detail in FIG. 3. The column sample and hold circuit combines the column pixel row operation (pixel reset, row select) and the column operation (the photo generation, photo sensing). The clamp signal activates the switch SW2 to place the capacitors of CS1 and CS2 in parallel for charging during the photo generation or conversion period of the light signal L to a light conversion electrical signal. The switch SW2 is the deactivated during the pixel reset time to provide the differential output signal. This combination causes the output voltage Vout to be equal to the differential voltage of pixel reset level and photo conversion electrical signal level, i.e., Vout=Vrst−Vsig of all the pixels in one row is stored in the column sample/hold circuit on series capacitors of CS1 and CS2 of each column. During the pixel readout, switch SW3 controlled by column select signal COL_SEL selects the column output. Column output drives the VIDEO AMP that applies the gain and offset correction to the output signal. The output of VIDEO AMP is the analog output that is digitized by an analog-to-digital converter (not shown). Since column bus has fairly large parasitic capacitance (CP), the pixel output Vout has been diluted. The actual input voltage to VIDEO AMP is given by:
                              V          IN                      VID            ⁢                                                  ⁢            AMP                          =                                            (                                                CS                  ⁢                                                                          ⁢                                      1                    ·                    CS                                    ⁢                                                                          ⁢                  2                                                                      CS                    ⁢                                                                                  ⁢                    1                                    +                                      CS                    ⁢                                                                                  ⁢                    2                                                              )                                                      (                                                      CS                    ⁢                                                                                  ⁢                                          1                      ·                      CS                                        ⁢                                                                                  ⁢                    2                                                                              CS                      ⁢                                                                                          ⁢                      1                                        +                                          CS                      ⁢                                                                                          ⁢                      2                                                                      )                            +              CP                                ·                      V            OUT                                              (        4        )                            Where:                    VINVID AMP is the voltage level representing the light level impinging upon the pixel being sensed.            CS1 is the capacitance value of the series capacitor CS1.            CS2 is the capacitance value of the series capacitor CS2.            CP is the capacitance value of the parasitic capacitor CP.Although the passive column output scheme dilutes the output voltage, the column fixed pattern noise (FPN) is very low.                        
An alternate approach for the column sample/hold circuits is implementing active column circuit. The active circuit in column sample/hold approach can eliminate the signal dilution due to charge sharing in passive readout scheme. The column fixed pattern introduced by active column circuit can be minimized by a double sampling scheme. FIG. 4 shows the schematic diagram of active column sample and hold approach.
In this approach, a source follower SF1 is placed between the node that develops the output voltage VOUT and the column select switch SW3. The source follower isolates the output voltage from the effects of the stray capacitor CP. This causes the actual input voltage to VIDEO AMP is given by:VINVID AMP=GVOUT  (5)                Where:                    G is the gain of source follower.                        
“Progress in Voltage and Current Mode On-Chip Analog-to-Digital converters for CMOS Image Sensors”, Panicacci, et al., Jan. 31, 1996, Found Jul. 13, 2004: http://techreports.jpl.nasa.gov/1996/1006.html describes CMOS active pixel sensors having row and column averaging circuits for varying the resolution of the image sensors.
“Variable Resolution CMOS Current Mode Active Pixel Sensor,” Coulombe, et al., Proceedings—The 2000 IEEE International Symposium on Circuits and Systems—ISCAS 2000, 2000, vol. 2, pp: 293-296, a current mediated active pixel sensor (APS) with variable image size and resolution for power saving, electronic zooming, and data reduction at the sensor level. The circuit can perform averaging of output signals in blocks of adjacent pixels (kernels) of size 1×1, 2×2 and 4×4, allowing data reduction without aliasing effects. To achieve this, a current approach is used, thus enabling high speed operation and low power supply capacity. The circuit compensates for pixel transconductance mismatch in addition to offset error via analog to digital conversion reference current scaling.
“Frame-Transfer CMOS Active Pixel Sensor with Pixel Binning”, Zhou, et al., IEEE Transactions on Electron Devices, October 1997 Vol.: 44, Issue: 10, pp.: 1764-1768, reports a first frame-transfer CMOS active pixel sensor (APS). The sensor architecture integrates an array of active pixels with an array of passive memory cells. Charge integration amplifier-based readout of the memory cells permits binning of pixels for variable resolution imaging.
U.S. Pat. No. 6,721,464 (Pain, et al.) discloses a high-speed on-chip windowed averaging system using photodiode-based CMOS imager. The system has an imager array, a switching network, computation elements, and a divider circuit. The imager array has columns and rows of pixels. The switching network is adapted to receive pixel signals from the image array. The plurality of computation elements operates to compute column and row averages.
U.S. Pat. No. 5,585,620 (Nakamura, et al.) teaches an image reading device (image scanner) that includes a resolution changing device. The resolution is changed by an averaging process circuit that averages the signals output from adjacent photoelectric sensor elements. The averaging process circuit changes a resolution of the image by a factor of m by averaging the signals output by m adjacent photoelectric sensor elements, where m is an integer.
U.S. Pat. No. 6,166,367 (Cho) describes a programmable arithmetic circuit to form multiple circuit modules for different arithmetic operations that share certain common electronic elements to reduce the number of elements. Such circuit can be integrated to an imaging sensor array such as a CMOS active pixel sensor array to perform arithmetic operations and analog-to-digital conversion for imaging processing such as pixel averaging for resolution reduction.
U.S. Pat. No. 6,104,844 (Alger-Meunier) teaches an image sensor that has adjustable resolution. Neighboring sensor elements are in each case combined into pixel sensor regions. During the recording of the image, the measured values of the sensor elements of each sensor region are averaged. In this case, each average value corresponds to a pixel of the recorded image. In this manner, production-dictated tolerances of the sensor elements are compensated for by the averaging.